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Early Momentum Metrics 
Leading Indicators for Community 
College Improvement 


By Clive R. Belfield, Davis Jenkins, and John Fink 


As community colleges across the country implement large-scale reforms to 
improve student success, they need timely and actionable metrics to determine if the changes 
they are making in a given year or term will likely improve student outcomes in the long run. In 
this brief, we examine how well nine measures of students’ progress in their first year predict 
student completion in subsequent years and thus how suitable these early momentum metrics, 
or EMMs, are as leading indicators of the effectiveness of institutional reforms. 


Based on analysis of student data from all community colleges in three = 

fa hate Mnied dict] f d a Based on analysis in 
states, we find that s do predict longer term success for students. We three states, we find that 
also find that a key factor in low completion rates, as well as in equity gaps measures of student 
in completion rates, is that many students do not gain early momentum progress in the first year 
in their first year. College outcomes would be substantially higher if more predict student completion 
students met EMMs. Our findings indicate the need for comprehensive in subsequent years. 


reforms to community college organization and practice to help more 
students gain early momentum on their way to earning a credential. 


First-Year Indicators of Longer-Term Success 


Throughout the nation, two-year and four-year colleges are recognizing that to “move the 
needle” on collegewide measures of success (e.g., completion rates) major redesign across the 
entire institution may be required to ensure that its programs, policies, and practices are well 
designed and aligned to promote student success. Hundreds of community colleges across 

the country are adopting whole-college reforms commonly referred to as “guided pathways” 
(Bailey, Jaggars, & Jenkins, 2015). As part of these reforms, colleges are mapping requirements 
and options for programs of study so that they are aligned with desired employment and 
transfer outcomes, redesigning the onboarding process to help new students explore options 
and interests and develop an academic completion plan, and revamping advising to better 
support students in making timely progress toward completion ofa credential. Faculty are also 
working together across disciplines to ensure that curricula and teaching prepare students to 
succeed in employment and transfer in their field of study. 


Because they involve a redesign of college practices at scale, guided pathways reforms take several 
years to implement (Jenkins, Lahr, Fink, & Ganga, 2018, pp. 2-3). Given that whole-college 
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reforms like guided pathways take so much time to 
carry out, lagging metrics of student performance, EMMs Used in the Current Analysis 
such as three- or six-year graduation rates, are not 
timely enough for college leaders to use to monitor and 
improve the reforms as they are being implemented. To 
formatively assess whether reforms being implemented 
in the present are likely to improve student outcomes 
several years hence, college leaders need leading metrics 
of longer-term success. These indicators need to be 
measurable over a relatively short time frame, and they 
should strongly predict success in college. 


Credit Momentum 
e Completed 6 or more college-level credits in 
the first semester (6 credits S1) 
e Completed 12 or more college-level credits in 
the first semester (12 credits S1) 
e Completed 15 or more college-level credits in 
the first year (15 credits Y1) 
e Completed 24 or more college-level credits in 
the first year (24 credits Y1) 
Previous research shows that students’ “momentum” e¢ Completed 30 or more college-level credits in 
in their first year is strongly associated with higher the first year (30 credits Y1) 
rates of completion in the longer term (Adelman, 
1999, 2006; Attewell, Heil, & Reisel, 2012; Wang, 
2017). Ifindeed first-year momentum is predictive of 
completion, college leaders can examine year-over-year 
trends in early momentum metrics for first-time 
entering student cohorts to see if the reforms they are 
implementing are likely to improve completion rates in 


Gateway Course Momentum 
e Completed college-level English in the first 
year (English Y1) 
e Completed college-level math in the first year 
(Math Y1) 
e Completed both college-level English and 
math in the first year (English & math Y1) 


the future. 

Here we use student transcript data from three Persistence Momentum 

community college state systems to assess the validity e Fall to spring persistence in the first year 
of early (first-year) momentum metrics, or EMMs, as (Persist S1 S2) 


leading indicators of student completion. In particular, 

we examine three sets of EMMs that previous research 

suggests should be predictive of completion. In addition 

to their potential usefulness as metrics for examining the effects of whole college reforms, these 
metrics may also help focus the attention of colleges implementing whole-college reforms 

on creating the early conditions for longer-term student success (Jenkins & Bailey, 2017). 
Conceptually, the metrics are grouped as follows: 


e Credit momentum metrics: Rates at which students complete a substantial number of college- 
level credits in their first year. These metrics focus colleges on reforms to new student 
advising, academic planning, and scheduling that help ensure that students are—from the 
start—making timely progress toward program completion. 


e Gateway course momentum metrics: Rates at which students take and pass college-level math 
and English courses in their first year. These metrics discourage colleges from relying on long 
sequences of prerequisite remediation courses, which too often fail to build students’ skills 
for college, and instead encourage them to integrate and contextualize academic support into 
college-level courses, an approach for which there is a growing body of evidence (see Ganga, 
Mazzariello, & Edgecombe, 2018). 


e Persistence momentum metric: Rate at which students are retained from the first to the second 
term. This metric encourages colleges to examine why so many students stop out before the 
end of the first term and to implement practices that help students to persist into the second 
term and beyond. 
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Student success can be measured in different ways. In this analysis, we use two measures that 
define student success in terms of credential completion: (1) completion of any community 
college award (including associate degrees and certificates) within three years of starting at 
the community college, and (2) completion of any award (including bachelor’s degrees) from 
any postsecondary institution (using matched data from the National Student Clearinghouse) 
within six years of starting at the community college. These measures capture the direct 
effectiveness of community colleges and how well these colleges prepare students for 
baccalaureate programs. Furthermore, given the potential role of EMMs as leading indicators 
of whether or not equity gaps will close in the longer term, we report analyses for two 
combined racial/ethnic subgroups of students to examine whether we find evidence that EMM 
attainment is predictive of longer-term success for these groups. We chose the first subgroup, 
Black/African American students and Hispanic/Latino students combined, as a proxy for 
historically underrepresented minority (URM) students, and the second subgroup, Asian 
students and White students combined, as a proxy for non-URM students.? 


Overall, for cohorts of community college students and for student racial/ethnic subgroups, 
we investigate: 


e How many students meet each EMM? 
e How strong are associations between EMMs and student success? 


e Whatare predicted outcomes if more students were to meet each EMM? 


Data From Three Community College Systems 


We use data from all community colleges across three anonymous state systems (X, Y, 

and Z). The dataset is composed of student transcripts, which include information on all 
postsecondary remedial and college-level courses taken and all credentials earned at each 
community college, as well as data on transfer and four-year degree completion. Transcripts 
also include student demographic (but not high school performance) information. The sample 
students are first-time entrants in 2010-12 at all colleges in the three state community college 
systems; these students are followed for up to six years after first enrollment. In total, the 
sample includes over 500,000 students across more than 75 colleges. 


EMM Attainment Rates 


Figure 1 shows the rates of EMM attainment across each college system. (For illustration, 

43 percent of students in System Y completed college-level English in year 1.) Interestingly, 
these attainment rates are very similar across the three systems. With the exception of the 
persistence EMM, far fewer than half of all students are “on track” in the sense that they have 
met any EMM. Roughly, about one in ten students are “on pace” in the sense that they have 
passed math and English gateway courses, or earned 12 credits in their first semester or 30 
credits in their first year. Even in terms of persistence, the attainment rates are far from ideal— 
about one third of students do not persist from their first to second semester. 
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Figure 1. 
EMM Attainment Rates Among All Students 
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Figure 2 shows the rates of EMM attainment for Black and Hispanic students only. (For 
illustration, 24 percent of Black and Hispanic students in System Y completed college-level 
English in year 1). Again, these rates are broadly similar across the three systems. However, 
compared with the results among all students, the rates of EMM attainment are much lower for 
Black and Hispanic students. Approximately one fifth of these students are “on track,” and only 
one in 20 are “on pace.” The persistence rate is about 60 percent, meaning that two fifths of 
Black and Hispanic students stay only one semester at community college. 


Figure 2. 
EMM Attainment Rates Among Black and Hispanic Students 
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Figure 3 shows the rates of EMM attainment among Black and Hispanic (URM) students 
relative to those among Asian and White (non-URM) students. There are sizeable gaps in rates 
across all of the EMMs. For example, in System X, Black and Hispanic students completed 
college-level math in their first year at half the rate of Asian and White students. For some 
EMMsand for some systems, Black and Hispanic students’ attainment rates are only one third 
those of Asian and White and students. Yet the gaps are not consistent across all EMMs or all 
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state systems. For example, for the EMM on earning 12 credits in the first semester, the relative 
rates of URM tonon-URM attainment in Systems X, Y, and Z are 48 percent, 24 percent, and 
80 percent, respectively. Hence, a range of different improvements may be needed in order for 
college systems to close gaps. 

Figure 3. 

EMM Attainment Rates: Relative Rates of URM to Non-URM Students 
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EMMs and Student Success 


Too few students succeed in college. Consistent with national figures, the three state systems 
report successful completion for substantially fewer than half of all students. Systems X, Y, and 
Z had three-year community college completion rates of 28, 20, and 25 percent, respectively. 
(These rates include completion of any community college degree or certificate, denoted as CC 
Award.) Tracking students’ attainment of any degree or certificate (including four-year degrees, 
denoted as Any Award) for six years, Systems X, Y, and Z had completion rates of 32, 22, and 
30 percent, respectively. We focus on how well EMMs predict these success rates. 


We estimate student success rates (three- and six-year completion rates, as described above) 
among those students who meet each EMM, controlling for other factors. These rates are 
adjusted from logit and propensity score matching estimators that account for student 
characteristics (gender, race/ethnicity, residence, intent, and socioeconomic status, as well 
as college, semester-entry, and year indicators). The estimates reported here are averages 
across sets of comparisons between students who meet each EMM and five different samples 
of students who do not meet the respective EMM. These five comparison samples are: all 
students, students who took English and math in their first year, students who ever took 
English and math, students who were designated college-ready in English, and students who 
were designated college-ready in math. The baseline rates indicate the average actual success 
rates for all students. For each estimate, we compare the success rate for students who meet the 
EMM against the baseline success rate. 
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Adjusted three-year CC Award rates are shown in Figure 4. These success rates are estimated 
for students who meet each EMM, controlling for student characteristics. (For example, 
controlling for student characteristics, 50 percent of students who earned 15 college-level 
credits in their first year completed a CC Award within three years [compared to the baseline 
rate of 28 percent] in System X.) 


Figure 4. 
Three-Year CC Award Rates Among EMM Attainment Students 
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Figure 5 shows the respective estimates for students who meet each EMM for Any Award. (For 
example, controlling for student characteristics, 61 percent of students who completed 15 
college-level credits in the first year completed Any Award within six years [compared to the 
baseline rate of 32 percent] in System X). 


Figure 5. 
Six-Year Any Award Rates Among EMM Attainment Students 


80% 


60% 


40% 
- | | | 
0% 


Baseline —_ English Math English 6credits 12credits Persist 15credits 24credits 30 credits 
& math S1 S1 $1S2 Y1 Y1 Y1 


Students who meet EMMs have substantially higher award rates. Broadly, the results are robust 
and consistent across methods, samples, and state systems. The magnitudes are very large, with 
most rates increasing over baseline rates by at least a factor of 0.5 and up to a factor of 3. And 
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the effects across EMMs are incremental: Meeting a “stricter” EMM is associated with a greater 
increase in award rates. 


The corresponding estimates of success for Black and Hispanic students combined are given 

in Figures 6 and 7. These results too are robust and consistent across methods, samples, and 
state systems. They show that Black and Hispanic students who meet EMMs have substantially 
higher CC Award and Any Award rates. The magnitudes are very large, and the effects are 

again incremental across EMMs. The similarity of these estimates to those of the full sample is 
important in light of the low EMM attainment rates generally. If Black and Hispanic students 
meet the EMMs, the estimated increase in their chances of completing an award are of a similar 
magnitude as those for Asian and White students who meet the EMMs. Together, these 
findings imply that a promising approach to improving completion rates for Black and Hispanic 
students, as for other students, is to help them to start strong and gain early momentum, as 
measured here by the EMMs. 


Figure 6. 
Three-Year CC Award Rates Among Black and Hispanic EMM Attainment Students 
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Figure 7. 
Six-Year Any Award Rates Among Black and Hispanic EMM Attainment Students 
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Overall, students who meet EMMs are more successful, controlling for characteristics such as 
race/ethnicity, gender, and socioeconomic status. The key issue is the rate of EMM attainment 
by students in the first year (not how students perform later in college). Thus, getting students 
to meet EMMs should boost success. 


Moreover, each type of EMM has independent—additive—effects. For example, students who 
complete gateway math or pass 15 credits in year 1 have greater success than students who do 
not meet these EMMs, but students who complete gateway math and pass 15 credits in year 1 
have even greater success. 


Simulated Outcomes for Meeting EMM Targets 


The predictive relationships can be used to model what would happen to longer-term success 
if more students met each EMM. To illustrate, we estimate predicted completion rates if 50 
percent more students meet each EMM. (For example, the gateway math EMM rate might 
increase from 10 to 15 percent, with all other EMM rates fixed. We do not simulate effects 
for persistence, as rates already exceed 50 percent.) These simulations require two key 
assumptions to keep in mind while interpreting results: (1) We assume that many more 
students will meet each EMM, and (2) we assume that the additional students meeting each 
EMM will complete college credentials at the same rate as the relatively smaller group of 
students meeting the EMM historically. Thus, these simulations are approximate—they 
illustrate the potential magnitude of change for substantially increasing EMM attainment. 


The simulation results for CC Award are shown in Figure 8. (For example, the CC Award rate 

is predicted to increase by approximately 5 percentage points in System Y if 50 percent more 
students completed 15 college-level credits in their first year.) If 50 percent more students met 
a particular EMM, the CC Award rates would increase by at least 1 percentage point, on average 
3 percentage points, and as high as 5 percentage points. Given baseline CC Award rates of 
20-28 percent, these are significant changes in college-level outcomes. 


Figure 8. 

Predicted Percentage Point Increase in Three-Year CC Award Rate If EMM Attainment Rate Increased by 50 Percent 
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The simulation results for Any Award are shown in Figure 9. (For example, the Any Award 
completion rate is predicted to increase approximately 6 percentage points in System Y if 50 
percent more students completed 15 college-level credits in the first year.) If 50 percent more 
students met a particular EMM, the Any Award rate would be approximately 3 percentage 
points higher on average. Again, these results suggest that if EMM rates increased by 50 
percent, we might expect substantial improvements in college-level outcomes. 


Figure 9. 


Predicted Percentage Point Increase in Six-Year Any Award Rate If EMM Attainment Rate Increased by 50 Percent 
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An alternative approach is to look at what changes would be needed to increase the Any Award 
rate by 10 percentage points. (Again, these simulations are subject to the two assumptions 
noted above, and persistence is not modelled.) These changes are shown in Figure 10. (For 
example, in order to increase the Any Award rate by 10 percentage points, the percentage of 
System Z students completing 15 college-level credits in their first year would need to double, 
or increase by a factor of 2.) Looking at each EMM, to increase the Any Award rate by 10 
percentage points would require significant increases in the rate of EMM attainment. For most 
EMMs, the rate of EMM attainment would need to double, holding the other EMM attainment 
rates constant. For the EMMs with the lowest rates of attainment (i.e., the “strictest” EMMs), 
such as passing both gateway English and math or completing 30 credits in the first year, the 
rate of EMM attainment would need to increase by a factor of about 4 or more. (While it might 
be reasonable to think that “stricter” EMMs should require less ofa factor increase to yield 

the same increase in completion rates, our simulations suggest that the opposite is the case. 
Because these EMMs have much lower rates of attainment [such as 10 percent of the cohort] 
than other EMMs, they need to be increased substantially in these simulations to have an equal 
impact on completion rates.) 
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Figure 10. 
Factor Increase in EMM Attainment Rate Needed to Increase Six-Year Any Award Rate by 10 Percentage Points 
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Overall, higher attainment across any of the EMMs is expected to increase student outcomes. 
However, colleges face a trade-off in which students to target. For some EMMs, increasing 
attainment by 50 percent will affect few students; yet these students will then be much more 
likely to graduate. For other EMMs, increasing attainment by 50 percent will affect many 
students; yet these students will be only modestly more likely to graduate. Thus, some EMMs 
affect fewer students, but they do so strongly; other EMMs affect more students, but the 
average effect is relatively weak. 


Conclusion 


Our investigation is motivated to validate leading indicators that colleges can track within 
students’ first year that strongly predict their eventual success and that colleges can therefore 
use to evaluate the likely longer-term effects of large-scale reforms. We examine nine early 
momentum metrics or EMMs that measure credit accumulation, gateway course completion, 
and persistence. These EMMs strongly predict completion rates at two- and four-year colleges. 
Our findings suggest that these EMMs are valid leading indicators of students’ future credential 
completion outcomes (at the starting institution or at another college). 


However, too few students are currently “on track” or “on pace” with regard to these EMMs. 
This is the case for students generally and especially for Black and Hispanic students, for whom 
gaps in EMM attainment are very large. Encouragingly, we observe that the EMMs strongly 
predict longer-term completion rates not only for students generally but also for Black and 
Hispanic students, suggesting that college efforts to close racial equity gaps in early momentum 
represent a crucial step toward closing gaps in credential completion. 


Community colleges have considerable scope for improving rates of EMM attainment and 
thereby promoting student success. To improve student success, colleges need to rethink 

how they teach and support students throughout their entire experience with the college. 

The findings from this brief highlight the importance of reforms to the early part of students’ 
experience. But even after meeting EMMs and thus making early progress, students still need 
to complete their programs. Hence, students need intensive guidance and academic support as 


EARLY MOMENTUM METRICS | CCRC RESEARCH BRIEF | JULY 2019 EEE 


they enter college and throughout their first year, but they also need sustained support anda 
clear path to completion. 


Many community colleges across the country are implementing guided pathways (GP) 
reforms to improve their effectiveness in helping students gain early momentum and sustain 
that momentum through to completion. GP involves a set of organizational changes that 
transform how students navigate through programs of study to earn credentials (see Bailey et 
al., 2015). GP offers colleges a framework for reform that is both comprehensive (addressing 
each step in students’ educational journey at college) and holistic (engaging cross-functional 
units within college, like student/academic affairs, transfer/workforce programs, and credit/ 
noncredit offerings). 


Essential features of GP reforms align with our analysis of how meeting EMMs influence 
success. First, GP affects how students connect with and enter college: Passing gateway courses 
gets students started on a guided pathway. Second, GP involves significant changes to first-year 
advisement and supports to enable students to earn more credits: Accumulating credits early 
on moves students forward ona path to completion. Finally, reforms to teaching and learning 
in gateway courses (including math and English but also Biology 


101, Accounting 101, and other introductory programmatic Colleges need information 
courses) are crucial to boosting other components of early to determine how successful 
momentum not explicitly addressed in this analysis, suchas whole-college reforms are 


likely to be in improving 


students’ aspirational or motivational momentum (Wang, 2017). : : 
completion rates following full 


Colleges need information to determine how successful implementation. The credit, 
whole-college reforms like GP are likely to be in improving gateway course, and persistence 
completion rates following full implementation. The credit, EMMs described here meet this 
gateway course, and persistence EMMs described here meet this need—they are early, leading 
need—they are early, leading indicators that strongly predict indicators that strongly predict 
student completion over a longer term. Furthermore, colleges student completion. 


are unlikely to close racial/ethnic and other equity gaps in 

longer-term completion rates if they do not first close these gaps in these leading indicators. 
By disaggregating subgroups and using EMMsas leading indicators, colleges can formatively 
assess what changes in practices are effective in the short term and are therefore likely to 
increase completion rates and close equity gaps in the longer term. Our analysis suggests that 
colleges that are able to implement practices that increase attainment of these EMMsare likely 
to see improved completion rates over the longer term. 


Endnotes 


1. Our use of these two categories—historically underrepresented minority (URM) students 
and non-URM students—is imprecise in that students from racial/ethnic groups other 
than the ones we include may be considered URM students and that persons from some 
Asian populations, such as Vietnamese, may be more aptly described as URM than 
non-URM students. 
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